Skip to content

feat(profiling): surface native-heap partition arming via dogstatsd gauge + warning - #19376

Closed
vlad-scherbich wants to merge 2 commits into
vlad/native-heap-phase2-ownership-partitionfrom
vlad/native-heap-arming-observability-pr-f
Closed

feat(profiling): surface native-heap partition arming via dogstatsd gauge + warning#19376
vlad-scherbich wants to merge 2 commits into
vlad/native-heap-phase2-ownership-partitionfrom
vlad/native-heap-arming-observability-pr-f

Conversation

@vlad-scherbich

@vlad-scherbich vlad-scherbich commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Stack: A — cdylib/build (#19078) ← B — activation (#19079) ← C — live-heap (#19325) ← Phase 2 — ownership partition (#19349) ← F — arming observability (this PR).

Prev: #19349 (base of this PR).

Description

Target 1 of the native-heap observability follow-ups: make the ownership-partition arming decision confirmable at runtime.

Today the decision is emitted only as a one-shot stdlib LOG.info(...) in profiler._start_service. In real deploys the app's structured JSON logger drops ddtrace stdlib records and pods/exec is RBAC-blocked, so armed=true|false is unconfirmable at runtime — which blocks Phase 2 A/B validation.

This surfaces the decision two ways at service start, gated on native-heap profiling being enabled so the default majority of processes (which never enable it) stay completely silent — no new WARNING/metric noise:

  • dogstatsd gauge profiling.native_heap.partition_armed — 1 when armed, 0 when not. Tags: domains:OBJ_MEM, size_threshold_bytes:512. Emitted via the canonical internal client (get_dogstatsd_client(agent_config.dogstatsd_url)), the same accessor runtime_metrics/writer use.
  • WARNING log — the existing INFO arming line is upgraded to LOG.warning(...) with the same fields (armed=<bool> size_threshold_bytes=512 domains=OBJ|MEM) so it survives deploys whose JSON loggers filter ddtrace INFO records.

Emission is one-shot at _start_service (never on a hot path) and fail-safe: if the dogstatsd client is unavailable/raises, the error is swallowed (LOG.debug) and arming/startup is unaffected.

Out of scope (separate follow-up): the DD_PROFILING_NATIVE_HEAP_ENABLED config registration, which collides on append-only config files.

Testing

tests/profiling/test_native_heap_gotter.py (run via the run-tests skill / riot profile venv, py3.13):

  • armed → gauge emitted once with value 1 + WARNING logged with armed=True
  • enabled but gotter not installed → gauge value 0 + WARNING with armed=False
  • dogstatsd client raising must not break arming or startup
  • existing arming/partition wiring tests updated for the new WARNING on stderr (only the arming line is allowed)
  • feature disabled (default) → no WARNING, no gauge (stderr unchanged)

Risks

Low. The WARNING/gauge only emit when DD_PROFILING_NATIVE_HEAP_ENABLED=true; the default path is byte-for-byte silent as before. The gauge is best-effort and cannot break profiler startup.

Additional Notes

No release note: native-heap profiling is behind unmerged/unreleased stacked PRs, so this is observability for a not-yet-released feature (changelog/no-changelog).

@vlad-scherbich vlad-scherbich added the changelog/no-changelog A changelog entry is not required for this PR. label Jul 29, 2026
@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Jul 29, 2026

Copy link
Copy Markdown

Codeowners resolved as

ddtrace/profiling/profiler.py                                           @DataDog/profiling-python
tests/profiling/test_native_heap_gotter.py                              @DataDog/profiling-python

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Jul 29, 2026 •
edited
Loading

Copy link
Copy Markdown

Circular import analysis

⚠️ Existing circular imports

There are 3 circular imports that already exist on the base branch and have not been changed by this PR.

ddtrace.trace -> ddtrace._trace.tracer -> ddtrace.internal.debug -> ddtrace.trace
ddtrace -> ddtrace.trace -> ddtrace._trace.tracer -> ddtrace.internal.debug -> ddtrace
ddtrace -> ddtrace.trace -> ddtrace._trace.tracer -> ddtrace.internal.debug -> ddtrace.internal.runtime.runtime_metrics -> ddtrace

@datadog-datadog-prod-us1-2

datadog-datadog-prod-us1-2 Bot commented Jul 29, 2026 •
edited by datadog-datadog-prod-us1 Bot
Loading

Copy link
Copy Markdown
Contributor

Pipelines

✨ Unblock PR with BitsAI

⚠️ Warnings

🚦 1 Pipeline job failed

DataDog/apm-reliability/dd-trace-py | validate_supported_configurations_v2_local_file   View in Datadog   GitLab

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 2f180af | Docs | Datadog PR Page | Give us feedback!

@vlad-scherbich
feat(profiling): surface native-heap partition arming via dogstatsd g…
8b3d607 
…auge + warning

The native-heap ownership-partition arming decision was only observable via a
one-shot stdlib INFO log, which real deploys drop (structured JSON loggers
filter ddtrace stdlib records) and pods/exec is RBAC-blocked, so armed=true|false
could not be confirmed at runtime.

Surface the decision two ways at service start (gated on native-heap being
enabled, so the default majority of processes stay silent):
- emit a dogstatsd gauge profiling.native_heap.partition_armed (1=armed, 0=not)
  via the canonical internal client; fail-safe if the client is unavailable
- upgrade the existing INFO arming line to WARNING (same fields)
@vlad-scherbich
refactor(profiling): add type annotations to native-heap arming obser…
2f180af 
…vability

Annotate the arming-observability additions: the
_NATIVE_HEAP_SIZE_THRESHOLD_BYTES / _NATIVE_HEAP_PARTITION_ARMED_METRIC
module globals and the _emit_native_heap_partition_armed_gauge local
(profiler.py), plus the locals in the new dogstatsd-gauge/WARNING tests.
@vlad-scherbich
vlad-scherbich force-pushed the vlad/native-heap-phase2-ownership-partition branch from 8c9f8bc to ba29a15 Compare July 29, 2026 21:05
@vlad-scherbich
vlad-scherbich force-pushed the vlad/native-heap-arming-observability-pr-f branch from fa3b058 to 2f180af Compare July 29, 2026 21:06
@vlad-scherbich

vlad-scherbich commented Jul 29, 2026

Copy link
Copy Markdown
Contributor Author

Closing: the deterministic ownership-handoff integration test (test_native_heap_ownership_handoff_end_to_end on #19349) proves Phase-2 partition arming without needing the dogstatsd gauge. Arming is also confirmable via eBPF on covered nodes. The stdlib INFO log was already invisible in JSON-logging setups; this metric was a workaround for that observability gap, superseded by the CI proof.

@vlad-scherbich

vlad-scherbich commented Jul 29, 2026

Copy link
Copy Markdown
Contributor Author

Closing rationale (recorded for the stack): superseded by the deterministic ownership-handoff integration test on #19349 (test_native_heap_ownership_handoff_end_to_end), which now runs in CI via the dedicated profiling_native_heap_gotter_e2e job. That test proves partition arming + dedup using the gotter test-support hook-hit counter, without needing a runtime dogstatsd gauge.

The arming observability metric was draft/pending keep-or-close; closing it as redundant with the CI proof. If we still want prod dogfooding visibility later, we can revisit as optional follow-up work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

changelog/no-changelog A changelog entry is not required for this PR.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@vlad-scherbich